Camera module and super resolution image processing method thereof

ABSTRACT

A camera module includes an image acquisition unit configured to acquire a plurality of image frames having a spatial phase difference therebetween, an image generation unit configured to generate image data having a resolution higher than a resolution of each of the plurality of image frames using the plurality of image frames, and a depth information extraction unit configured to extract depth information about an object using the image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/971,907, filed Aug. 21, 2020; which is the U.S. national stageapplication of International Patent Application No. PCT/KR2019/002163,filed Feb. 21, 2019, which claims the benefit under 35 U.S.C. § 119 ofKorean Patent Application No. 10-2018-0022121, filed Feb. 23, 2018, thedisclosures of each of which are incorporated herein by reference intheir entirety.

TECHNICAL FIELD

Embodiments relate to a camera module and a super-resolution imageprocessing method thereof.

BACKGROUND ART

People who use portable devices demand optical devices that have highresolution, are small, and have various photographing functions (anoptical zoom-in/zoom-out function, an auto-focusing (AF) function, ahand-tremor compensation or optical image stabilizer (OIS) function,etc.). Such photographing functions may be realized by directly moving aplurality of lenses that are combined. In the case in which the numberof lenses is increased, however, the size of an optical device mayincrease.

The auto-focusing and hand-tremor compensation functions are performedby moving or tilting a plurality of lens modules, which are fixed to alens holder in the state in which the optical axes thereof are aligned,along an optical axis or in a direction perpendicular to the opticalaxis, and a separate lens moving apparatus is used to move the lensmodules. However, the lens moving apparatus has high power consumption,and an additional cover glass needs to be provided separately from thecamera module in order to protect the lens moving apparatus, thusleading to an increase in the overall thickness of a device.

Further, with increasing user demand for a high-quality image, a cameramodule capable of providing a super-resolution image is required. Inorder to generate a super-resolution image, however, the number ofpixels included in an image sensor is inevitably increased, which mayresult in an increase in the size of the image sensor and increasedpower consumption. Here, “super resolution (SR)” means conversion ofimage information having a given low resolution (LR) into imageinformation having a high resolution (HR).

In order to extract the depth of a pixel corresponding to an objectincluded in an image, a time of flight (ToF) method is used as onemethod of actively extracting a depth by radiating light onto an object.The ToF method is a method of radiating light onto an object andmeasuring the time taken for the light to return. A point spreadfunction (PSF) for image data is optimized and very simple, whereas aPSF for extracting depth information needs to be newly defined andoptimized.

DISCLOSURE Technical Problem

Embodiments provide a camera module and a super-resolution imageprocessing method thereof capable of simply extracting depth informationof a pixel at a high speed while providing a high-resolution image.

The objects to be accomplished by the disclosure are not limited to theabove-mentioned objects, and other objects not mentioned herein will beclearly understood by those skilled in the art from the followingdescription.

Technical Solution

A camera module according to an embodiment may include an imageacquisition unit configured to acquire a plurality of image frameshaving a spatial phase difference therebetween, an image generation unitconfigured to generate image data having a resolution higher than aresolution of each of the plurality of image frames using the pluralityof image frames, and a depth information extraction unit configured toextract depth information about an object using the image data.

For example, the image acquisition unit may include an optical unitconfigured to change a path along which light for an object travels, animage sensor configured to sense light incident along different paths,and a controller configured to control the optical unit and the imagesensor. The plurality of image frames may correspond to the resultssensed in sequence by the image sensor.

For example, the image acquisition unit may include a plurality ofoptical units having respectively different paths along which light foran object travels and a plurality of image sensors configured to senselight incident through the plurality of optical units. The plurality ofimage frames may correspond to the results sensed by the plurality ofimage sensors.

For example, the image generation unit may generate the image datahaving an intensity below.

$x_{\phi} = {\sum\limits_{k = 1}^{p}\left( {{D^{- 1}B_{k}^{- 1}A_{k}} - n_{k}} \right)}$

Here, x_(ϕ) represents the intensity of the image data, 1≤k≤p, prepresents the number of the image frames used to generate the imagedata, ϕ represents the degree of phase delay, D⁻¹ represents the inversematrix of D, D represents the size of a pixel of the image sensor, B_(K)⁻¹ represents the inverse matrix of B_(K), B_(K) represents opticalcharacteristics with respect to the depth information, n_(K) representsa noise component of a k^(th) image frame among p image frames, andA_(K) represents the intensity of the k^(th) image frame among p imageframes, and is as follows.A _(k) =DB _(k) x _(ϕ) +n _(k)

For example, the depth information extraction unit may calculate thedepth information as follows.

$x = {\tan^{- 1}\frac{x_{\phi 1} - x_{\phi 3}}{x_{\phi 2} - x_{\phi 3}} \times \frac{c}{4\pi f}}$

Here, x represents depth information, c represents a luminous flux, andf represents a frequency.

A super-resolution image processing method of a camera module accordingto another embodiment may include (a) acquiring a plurality of imageframes having a spatial phase difference therebetween, (b) generatingimage data having a resolution higher than the resolution of each of theplurality of image frames using the plurality of image frames, and (c)extracting depth information about an object using the image data.

For example, step (a) may include changing a path along which light foran object travels and sensing light incident along different paths insequence to acquire the plurality of image frames.

For example, step (a) may include sensing light for an objectsimultaneously in different paths to acquire the plurality of imageframes.

For example, step (b) may include obtaining the image data having anintensity below.

$x_{\phi} = {\sum\limits_{k = 1}^{p}\left( {{D^{- 1}B_{k}^{- 1}A_{k}} - n_{k}} \right)}$

Here, x_(ϕ) represents the intensity of the image data, 1≤k≤p, prepresents the number of the image frames used to generate the imagedata, ϕ represents the degree of phase delay, D⁻¹ represents the inversematrix of D, D represents the size of a pixel of an image sensorobtaining each of the image frames, B_(K) ⁻¹ represents the inversematrix of B_(K), B_(K) represents optical characteristics with respectto the depth information, n_(K) represents a noise component of a k^(th)image frame among p image frames, and A_(K) represents the intensity ofthe k^(th) image frame among the p image frames, and is as follows.A _(k) =DB _(k) x _(ϕ) +n _(k)

For example, step (c) may include obtaining the depth information asfollows.

$x = {\tan^{- 1}\frac{x_{\phi 1} - x_{\phi 3}}{x_{\phi 2} - x_{\phi 3}} \times \frac{c}{4\pi f}}$

Here, x represents depth information, c represents a luminous flux, andf represents a frequency.

For example, the super-resolution image processing method of a cameramodule may further include calibrating the depth information.

The above aspects of the present disclosure are only a part of theexemplary embodiments of the present disclosure, and various embodimentsbased on technical features of the present disclosure may be devised andunderstood by those skilled in the art from the following detaileddescription of the present disclosure.

Advantageous Effects

According to a camera module and a super-resolution image processingmethod thereof according to an embodiment of present disclosure, it ispossible to simply extract depth information at a high speed using asmall amount of computation while providing a super-resolution image.

The effects achievable through the disclosure are not limited to theabove-mentioned effects, and other effects not mentioned herein will beclearly understood by those skilled in the art from the followingdescription.

DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a flowchart for explaining a super-resolution imageprocessing method of a camera module according to an embodiment.

FIG. 2 illustrates a schematic block diagram of a camera moduleaccording to an embodiment for performing the super-resolution imageprocessing method shown in FIG. 1 .

FIG. 3 illustrates a block diagram of an embodiment of the imageacquisition unit shown in FIG. 2 .

FIG. 4 illustrates a cross-sectional view of an embodiment of the cameramodule including the image acquisition unit shown in FIG. 3 .

FIG. 5 illustrates a block diagram of another embodiment of the imageacquisition unit shown in FIG. 2 .

FIG. 6 is a diagram for explaining an operation method of an embodimentof the image acquisition unit.

FIG. 7 is a diagram for explaining the operation method of the imageacquisition unit explained in FIG. 6 in more detail.

FIG. 8 is a timing diagram of the operation method of the camera moduleaccording to an embodiment.

FIG. 9 is a flowchart for explaining an embodiment of a calibration stepin the super-resolution image processing method according to theembodiment.

FIGS. 10(a) and (b) are waveform diagrams for helping understanding ofthe calibration step shown in FIG. 9 .

FIG. 11(a) illustrates raw data, and FIG. 11(b) illustrates theintensity of an electric charge sensed and output by an image sensor.

FIGS. 12(a) to (c) are diagrams for explaining a super-resolution imageprocessing method according to a comparative example.

FIGS. 13(a) to (c) are diagrams for explaining the super-resolutionimage processing method according to the embodiment.

BEST MODE

Hereinafter, exemplary embodiments will be described in detail withreference to the accompanying drawings. While the disclosure is subjectto various modifications and alternative forms, specific embodimentsthereof are shown by way of example in the drawings. However, thedisclosure should not be construed as being limited to the embodimentsset forth herein, but on the contrary, the disclosure is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the embodiments.

It may be understood that, although the terms “first”, “second”, etc.may be used herein to describe various elements, these elements are notto be limited by these terms. These terms are generally only used todistinguish one element from another. In addition, terms particularlydefined in consideration of the construction and operation of theembodiments are used only to describe the embodiments, but do not definethe scope of the embodiments.

In the following description of the embodiments, it will be understoodthat, when each element is referred to as being “on” or “under” anotherelement, it can be directly on or under another element or can beindirectly formed such that one or more intervening elements are alsopresent. In addition, when an element is referred to as being “on” or“under”, “under the element” as well as “on the element” may be includedbased on the element.

In addition, relational terms, such as “on/upper part/above” and“under/lower part/below”, are used only to distinguish between onesubject or element and another subject or element without necessarilyrequiring or involving any physical or logical relationship or sequencebetween such subjects or elements.

Hereinafter, a super-resolution image processing method of a cameramodule according to an embodiment will be described with reference tothe accompanying drawings.

FIG. 1 illustrates a flowchart for explaining a super-resolution imageprocessing method of a camera module according to an embodiment, andFIG. 2 illustrates a schematic block diagram of a camera moduleaccording to an embodiment for performing the super-resolution imageprocessing method shown in FIG. 1 .

Although the super-resolution image processing method shown in FIG. 1will be described as being performed in the camera module shown in FIG.2 , the embodiment is not limited thereto. That is, the super-resolutionimage processing method shown in FIG. 1 may be performed in a cameramodule having a configuration different from that of the camera moduleshown in FIG. 2 , and the camera module shown in FIG. 2 may perform asuper-resolution image processing method different from that shown inFIG. 1 .

The camera module shown in FIG. 2 may include an image acquisition unit100, an image generation unit 200, and a depth information extractionunit 300.

The image acquisition unit 100 shown in FIG. 2 acquires a plurality ofimage frames, which have a spatial phase difference corresponding to asubpixel interval therebetween, and outputs the acquired plurality ofimage frames to the image generation unit 200 (step 10). When thedistance between pixels (e.g. the distance between the centers ofpixels) is defined as 1 pixel distance (PD), a half thereof correspondsto 0.5 PD. The aforementioned subpixel interval may be 0.5 PD, but theembodiment is not limited thereto.

FIG. 3 illustrates a block diagram of an embodiment 100A of the imageacquisition unit 100 shown in FIG. 2 .

The image acquisition unit 100A shown in FIG. 3 may include an opticalunit 110, an image sensor 120, and a controller 130.

The optical unit 110 may change the path along which light for an objecttravels under the control of the controller 130.

The image sensor 120 may, under the control of the controller 130, senselight beams incident along different paths and output the results ofsensing to the image generation unit 200 as image frames through anoutput terminal OUT1. The image sensor 120 sequentially senses lightbeams incident along different paths. Accordingly, the results sensed insequence by the image sensor 120 may correspond to a plurality of imageframes having a spatial phase difference therebetween.

The controller 130 may control the optical unit 110 and the image sensor120. In particular, the controller 130 may change the path along whichlight travels from the optical unit 110 so that a plurality of imageframes, sequentially sensed and output by the image sensor 120, has aspatial phase difference corresponding to a subpixel intervaltherebetween.

Hereinafter, embodiments of the camera module including the imageacquisition unit 100A shown in FIG. 3 will be described using theCartesian coordinate system. However, other coordinate systems may beused. In the Cartesian coordinate system, an x-axis, a y-axis, and az-axis are perpendicular to each other, but the embodiments are notlimited thereto. That is, the x-axis, the y-axis, and the z-axis mayintersect each other obliquely.

FIG. 4 illustrates a cross-sectional view of an embodiment of the cameramodule including the image acquisition unit 100A shown in FIG. 3 .

Referring to FIG. 4 , the camera module may include a lens assembly, animage sensor 120, and a main board 132. Here, the lens assembly maycorrespond to an embodiment of the optical unit 110 shown in FIG. 3 ,and may include a lens barrel 112, a holder 114, a first lens L1, and asecond lens L2. At least one of these components may be omitted, or thevertical arrangement of these components may be changed.

The lens assembly may transmit light incident thereon from the outsideof the camera module so that an optical signal is transmitted to theimage sensor 120. The lens assembly may include at least one lens. Thelenses included in the lens assembly may form one optical system, andmay be aligned along the optical axis of the image sensor 120.

The lens barrel 112 may be coupled to the holder 114, and may include aspace formed therein to accommodate the first lens L1 and the secondlens L2. The lens barrel 112 may be engaged with the first lens L1 andthe second lens L2 in a rotational engagement manner, but this is merelyexemplary. These components may be engaged in any of other manners, forexample, using an adhesive.

The first lens L1 may be disposed in front of the second lens L2. Thefirst lens L1 may be composed of at least one lens, or two or morelenses may be aligned along the central axes thereof to form an opticalsystem. Here, the central axis may be the same as the optical axis ofthe optical system of the camera module. The first lens L1 may becomposed of one lens, as shown in FIG. 4 , but the disclosure is notnecessarily limited thereto.

The second lens L2 may be disposed behind the first lens L1. Lightincident on the first lens L1 from the outside of the camera module maypass through the first lens L1 and may be incident on the second lensL2. The second lens L2 may be composed of at least one lens, or two ormore lenses may be aligned along the central axes thereof to form anoptical system. Here, the central axis may be the same as the opticalaxis of the optical system of the camera module. The second lens L2 maybe composed of one lens, as shown in FIG. 4 , but the disclosure is notnecessarily limited thereto.

The first lens L1 and the second lens L2 may be referred to as a ‘firstsolid lens’ and a ‘second solid lens’, respectively, in order to bedistinguished from a liquid lens.

In FIG. 4 , the lens assembly is illustrated as including two lenses L1and L2, but the embodiment is not limited thereto. In anotherembodiment, the lens assembly may include only one lens, or may includethree or more lenses.

The holder 114 serves to accommodate and support at least one lens. Theholder 114 may be coupled to the lens barrel 112 to support the lensbarrel 112, and may be coupled to the main board 132 to which the imagesensor 120 is attached.

The holder 114 may have a spiral structure, and may be engaged with thelens barrel 112 having a spiral structure as well in a rotationalengagement manner. However, this is merely exemplary. The holder 114 andthe lens barrel 112 may be engaged with each other using an adhesive(e.g. an adhesive resin such as epoxy), or the holder 114 and the lensbarrel 112 may be integrally formed with each other.

The image sensor 120 corresponds to the image sensor 120 shown in FIG. 3. The image sensor 120 may be mounted on the main board 132, and mayinclude a pixel array configured to receive an optical signal, havingpassed through the lens assembly, and to convert the optical signal intoan electrical signal corresponding thereto, a driving circuit configuredto drive a plurality of pixels included in the pixel array, and areadout circuit configured to read an analog pixel signal of each pixel.The readout circuit may compare the analog pixel signal with a referencesignal, and may generate a digital pixel signal (or an image signal)through analog-to-digital conversion. Here, the digital pixel signal ofeach of the pixels included in the pixel array constitutes an imagesignal, and the image signal may be transmitted in a frame unit and thusmay be defined as an image frame. That is, the image sensor may output aplurality of image frames.

The main board 132 may be disposed under the holder 114 and may includewires for transmitting an electrical signal between the respectivecomponents together with the controller 130. In addition, a connector(not shown) for realizing electrical connection with a power source orother devices (e.g. an application processor) present outside the cameramodule may be connected to the main board 132.

The main board 132 may be configured as a rigid flexible printed circuitboard (RFPCB) and may be bent depending on the requirements of the spacein which the camera module is mounted, but the embodiment is not limitedthereto.

In addition, the camera module may further include a filter 116 fortransmitting or blocking infrared (IR) light. To this end, the filter116 may be implemented as a glass. The filter 116 may filter lightwithin a specific wavelength range among light beams that have passedthrough the second lens L2. The filter 116 may be mounted and fixed in arecess formed in the holder 114. To this end, the holder 114 may includetherein a space in which the filter 116 may be attached thereto underthe lens barrel 112.

The above-described camera module shown in FIGS. 3 and 4 may change theoptical path through various methods.

According to an embodiment, at least one lens included in the lensassembly may include a variable lens. The variable lens may change theoptical path of the lens assembly under the control of the controller130. The variable lens may change the optical path of light incident onthe image sensor 120, and may change, for example, the focal length ofan optical signal, the angle of a field of view (FOV), or the directionof the FOV. For example, the variable lens may be configured as a liquidlens or a variable prism. The variable lens may be composed of at leastone lens and an actuator engaged with the at least one lens. Here, theat least one lens may be a liquid lens or a solid lens. The actuator maycontrol the physical displacement of the at least one lens engagedtherewith under the control of the controller 130. That is, the actuatormay adjust the distance between the at least one lens and the imagesensor 120, or may adjust the angle between the at least one lens andthe image sensor 120. Alternatively, the actuator may shift the at leastone lens in the x-axis and y-axis directions of the plane formed by thepixel array of the image sensor 120. In addition, the actuator may serveto change the optical path of light incident on the pixel array of theimage sensor 120. For example, when a liquid lens is not included in theat least one lens included in the variable lens, that is, when the atleast one lens included in the variable lens is a solid lens, theactuator may shift the at least one lens in at least one of the verticaldirection or the horizontal direction in response to a first controlsignal C1 output from the controller 130.

The variable lens may be disposed at any one of first to fourthpositions P1 to P4. However, this is merely exemplary, and the variablelens may be located elsewhere depending on the presence or absence ofthe first lens L1, the second lens L2, and the filter 116 or dependingon the relative positions thereof. However, the variable lens may belocated on the optical path, which is a region through which lightincident on the lens assembly passes, and may change the focal length orthe FOV angle. The first position P1 is a position corresponding to theoutside of the lens barrel 112, and the second position P2 is a positioncorresponding to a region above the first lens L1 within the lens barrel112. The third position P3 is a position corresponding to a regionbetween the first lens L1 and the second lens L2 within the lens barrel112, and the fourth position P4 is a position corresponding to a regionbelow the second lens L2 within the lens barrel 112.

Alternatively, according to another embodiment, the lens barrel 112, theholder 114, or the filter 116 may be shiftedupwards/downwards/leftwards/rightwards by the actuator (not shown) underthe control of the controller 130, whereby the optical path of lightincident on the image sensor 120 may be changed, and for example, thefocal length of an optical signal, the angle of a field of view (FOV),or the direction of the FOV may be changed.

Hereinafter, the operation of changing the FOV angle of the optical unit110 will be described with reference to FIG. 4 .

Referring to FIG. 4 , the lens assembly may have a specific field ofview (FOV). The FOV may refer to a range of incident light within whichthe image sensor 120 is capable of performing a capture operationthrough the lens assembly, and may be defined as an FOV angle. The FOVangle of a typical lens assembly may range from 60° to 140°. On thebasis of the x-axis and the y-axis defined when the lens assembly isviewed from above (i.e. from the direction perpendicular to the opticalaxis), the FOV angle may include a first FOV angle Fx and a second FOVangle Fy. The first FOV angle Fx refers to the angle of the FOV that isdetermined along the x-axis, and the second FOV angle Fy refers to theangle of the FOV that is determined along the y-axis.

A plurality of pixels included in the pixel array of the image sensor120 may be arranged in the form of an N×M matrix (where each of N and Mis an integer of 1 or more). That is, N pixels may be disposed along thex-axis, and M pixels may be disposed along the y-axis. An optical signalincident through the FOV corresponding to the first FOV angle Fx and thesecond FOV angle Fy is incident on the N×M pixel array.

The optical path of light passing through the lens assembly or the FOVof the lens assembly may be changed by a control signal C1. The controlsignal C1 may individually change the first FOV angle Fx and the secondFOV angle Fy. The changes in the first FOV angle Fx and the second FOVangle Fy according to the control signal C1 are determined by a firstangle variation θI_x and a second angle variation θI_y.

The first angle variation θI_x and the second angle variation θI_y maybe defined by the following Equation 1 and Equation 2, respectively.

$\begin{matrix}{{\frac{Fx}{N} \times a} \prec {\theta I\_ x} \prec {\frac{Fx}{N} \times b}} & \left\lbrack {{Equation}1} \right\rbrack\end{matrix}$ $\begin{matrix}{{\frac{Fy}{M} \times a} \prec {\theta I\_ y} \prec {\frac{Fy}{M} \times b}} & \left\lbrack {{Equation}2} \right\rbrack\end{matrix}$

Here, a may have a value greater than 0.1 and less than 0.5, and b mayhave a value greater than 1 and less than 2. However, the scope of theembodiment is not limited thereto.

In this case, θI_x and θI_y are angle variations with respect to animage generated by the image sensor 120, which are caused by the changein the optical path by the optical unit 110. The actual angle by whichthe optical unit 110 changes the optical path may be greater or lessthan the above angle variations.

However, the camera module and the super-resolution image processingmethod thereof according to the embodiment are not limited to anyspecific configuration or method in which the optical unit 110 changesthe optical path.

FIG. 5 illustrates a block diagram of another embodiment 100B of theimage acquisition unit 100 shown in FIG. 2 .

The image acquisition unit 100B shown in FIG. 5 may simultaneously senselight beams for an object along different paths to acquire a pluralityof image frames. To this end, the image acquisition unit 100B mayinclude first to H^(th) optical units 110-1 to 110-H and first to H^(th)image sensors 120-1 to 120-H. Here, H is a positive integer of 2 ormore.

Each of the first to H^(th) optical units 110-1 to 110-H forms a pathalong which light for an object travels. In this case, the paths alongwhich light beams travel through the first to H^(th) optical units 110-1to 110-H are different from each other.

The first to H^(th) image sensors 120-1 to 120-H sense respective lightbeams incident thereon through the first to H^(th) optical units 110-1to 110-H, and output the results of sensing to the image generation unit200 through an output terminal OUT2. The sensing results simultaneouslyoutput to the image generation unit 200 through the output terminal OUT2may correspond to a plurality of image frames having a spatial phasedifference corresponding to a subpixel interval therebetween.

After step 10, the image generation unit 200 may generate image datahaving a resolution higher than the resolution of each of the pluralityof image frames, acquired by the image acquisition unit 100, using theplurality of image frames acquired by the image acquisition unit 100,and may output the generated result to the depth information extractionunit 300 (step 20).

Hereinafter, step 20 and an embodiment of the image acquisition unit 100will be described.

FIG. 6 is a diagram for explaining an operation method of an embodimentof the image acquisition unit 100. FIG. 7 is a diagram for explainingthe operation method of the image acquisition unit 100 explained in FIG.6 in more detail.

FIG. 6 illustrates a mimetic diagram of a method of obtaining asuper-resolution image using a plurality of image frames having aspatial phase difference therebetween.

The pixel array of the image sensor 120 (120-1 to 120-H) may include aplurality of pixels arranged in the form of an N×M matrix. Forconvenience of description, the following description will be made onthe assumption that the pixel array includes a plurality of pixels (A1to A4) arranged in the form of a 2×2 matrix, as shown in FIG. 6 .

Each of the pixels A1 to A4 may generate image information (i.e. ananalog pixel signal corresponding to the optical signal) about each ofpixel scenes PS1 to PS4 using the optical signal transmitted through thelens assembly.

When the distance between pixels adjacent to each other in the x-axisdirection (or the y-axis direction) (e.g. the distance between thecenters of the pixels) is 1 pixel distance (PD), a half thereofcorresponds to 0.5 PD. Hereinafter, first to fourth pixel shifts A to Dwill be defined.

The first pixel shift A is to shift the respective pixels A1 to A4 by0.5 PD rightwards in the +x-axis direction, and B1 to B4 denote thepixels after completion of the first pixel shift A.

The second pixel shift B is to shift the respective pixels B1 to B4 by0.5 PD downwards in the +y-axis direction, and C1 to C4 denote thepixels after completion of the second pixel shift B.

The third pixel shift C is to shift the respective pixels C1 to C4 by0.5 PD leftwards in the −x-axis direction, and D1 to D4 denote thepixels after completion of the third pixel shift C.

The fourth pixel shift D is to shift the respective pixels D1 to D4 by0.5 PD upwards in the −y-axis direction, and A1 to A4 denote the pixelsafter completion of the fourth pixel shift D.

Here, the pixel shift functions not to shift the physical positions ofthe pixels of the pixel array, but to change the path along which lighttravels, as shown in FIG. 3 , or means an operation in which lighttravels through the plurality of optical units 110-1 to 110-H havingrespectively different optical paths, as shown in FIG. 5 , so that avirtual pixel (e.g. B1) between two pixels (e.g. A1 and A2) may acquirea pixel scene.

Referring to FIG. 7 , the respective pixels A1 to A4 may acquire a pixelscene S1, and the image sensor 120 (120-1 to 120-H) may generate a firstframe F1 from pixel signals of the respective pixels A1 to A4.

In response to the control signal C1 for changing the optical path orthe FOV rightwards by the first angle variation θI_x in order to realizethe first pixel shift A, the optical unit 110 shown in FIG. 3 may changethe optical path or the FOV of the lens assembly shown in FIG. 4rightwards by the first angle variation θI_x, whereby the first pixelshift A may be performed. Alternatively, in order to realize the firstpixel shift A, the optical path or the FOV of the first and secondoptical units 110-1 and 110-2 shown in FIG. 5 may have a differenceequivalent to the first angle variation θI_x therebetween. Thereafter,the respective pixels B1 to B4 may acquire a pixel scene S2, and theimage sensor 120 (120-1 to 120-H) may generate a second image frame F2from pixel signals of the respective pixels B1 to B4.

In response to the control signal C1 for changing the optical path orthe FOV downwards by the second angle variation θI_y in order to realizethe second pixel shift B, the optical unit 110 shown in FIG. 3 maychange the optical path or the FOV of the lens assembly shown in FIG. 4downwards by the second angle variation θI_y, whereby the second pixelshift B may be performed. Alternatively, in order to realize the secondpixel shift B, the optical path or the FOV of the second and thirdoptical units 110-2 and 110-3 shown in FIG. 5 may have a differenceequivalent to the second angle variation θI_y therebetween. Thereafter,the respective pixels C1 to C4 may acquire a pixel scene S3, and theimage sensor 120 (120-1 to 120-H) may generate a third image frame F3from pixel signals of the respective pixels C1 to C4.

In response to the control signal C1 for changing the optical path orthe FOV leftwards by the first angle variation θI_x in order to realizethe third pixel shift C, the optical unit 110 shown in FIG. 3 may changethe optical path or the FOV of the lens assembly shown in FIG. 4leftwards by the first angle variation θI_x, whereby the third pixelshift C may be performed. Alternatively, in order to realize the thirdpixel shift C, the optical path or the FOV of the third and fourthoptical units 110-3 and 110-4 shown in FIG. 5 may have a differenceequivalent to the second angle variation θI_x therebetween. Thereafter,the respective pixels D1 to D4 may acquire a pixel scene S4, and theimage sensor 120 (120-1 to 120-H) may generate a fourth image frame F4from pixel signals of the respective pixels D1 to D4.

In response to the control signal C1 for changing the optical path orthe FOV upwards by the second angle variation θI_y in order to realizethe fourth pixel shift D, the optical unit 110 shown in FIG. 3 maychange the optical path or the FOV of the lens assembly shown in FIG. 4upwards by the second angle variation θI_y, whereby the fourth pixelshift D may be performed. Alternatively, in order to realize the fourthpixel shift D, the optical path or the FOV of the fourth and firstoptical units 110-4 and 110-1 shown in FIG. 5 may have a differenceequivalent to the second angle variation θI_y therebetween. Thereafter,the respective pixels A1 to A4 may acquire a pixel scene S1, and theimage sensor 120 (120-1 to 120-H) may generate a fifth image frame F5from pixel signals of the respective pixels A1 to A4. Subsequently, thepixel shift and the generation of the frame through the shifted pixelsmay be repeatedly performed.

Here, each of the first angle variation θI_x and the second anglevariation θI_y may store information related to the extent to which theoptical path is changed so that the pixels are shifted by 0.5 PD, andmay be calculated in advance based on the first FOV angle Fx and thesecond FOV angle Fy and may be stored (e.g. by the image sensor 120 orthe controller 130).

The image sensor 120 shown in FIG. 3 may include a first region and asecond region, and the controller 130 may output the control signal C1to control the optical unit 110 such that the optical path of light,which is incident from the outside and passes through the lens assembly,is changed from the first region to the second region of the imagesensor 120. In addition, the image sensor 120 may further include athird region and a fourth region, and the controller 130 may output thecontrol signal C1 to control the optical unit 110 such that the opticalpath is changed from the second region to the third region of the imagesensor 120, and may output the control signal C1 to control the opticalunit 110 such that the optical path is changed from the third region tothe fourth region. The control signal C1 may include a signal forchanging the field of view (FOV) of the lens assembly in a firstdirection, a signal for changing the FOV of the lens assembly in asecond direction, a signal for changing the FOV of the lens assembly ina third direction, and a signal for changing the FOV of the lensassembly in a fourth direction.

The image generation unit 200 may synthesize the first to fourth imageframes and may generate an image acquired by a 2N×2M pixel array ratherthan by an N×M pixel array. As a method in which the image generationunit 200 synthesizes the first to fourth image frames, a method ofsimply merging the first to fourth image frames according to thepositions of the respective pixels (e.g. in the case of the first row,generating one frame by arranging the pixel signal of A1, the pixelsignal of B1, the pixel signal of A2, and the pixel signal of B2) or amethod of correcting the pixel signal of any one pixel (e.g. C1) usingthe pixel signals of the pixels adjacent thereto (e.g. A1, B1, A2, D1,D2, A3, B3, and A4) based on the principle in which the pixel scenes ofadjacent pixels overlap each other may be used. However, the scope ofthe embodiment is not limited thereto. Any of various super-resolutionimage generation methods may be used.

The image generation unit 200 may be referred to as a postprocessor. Thepostprocessor may generate a first super-resolution image frame bysynthesizing some of the plurality of image frames transmitted from theimage sensor 120 (120-1 to 120-H), and may then generate a secondsuper-resolution image frame by synthesizing the remaining ones of theplurality of image frames output from the image sensor 120 (120-1 to120-H).

According to the processing shown in FIGS. 6 and 7 , an image having aquadruple resolution may be generated by synthesizing a plurality ofimage frames acquired through pixel shift.

FIG. 8 is a timing diagram of the operation method of the camera moduleaccording to an embodiment.

Referring to FIG. 8 , the controller 130 may transmit a feedback signal,which indicates that the fourth pixel shift D has been completed by theoptical unit 110 in response to the control signal C1, to the imagesensor 120 as a control signal C2. In this case, the controller 130 maydetermine the completion of the fourth pixel shift D based on a responsesignal from the lens assembly or a separate timer. The respective pixelsA1 to A4 of the image sensor 120 that receives the feedback signal mayacquire the pixel scene S1, and the image sensor 120 may generate thefirst image frame F1 from the pixel signals of the respective pixels A1to A4. In the same manner, the second to fifth image frames F2 to F5 maybe generated. Subsequently, the pixel shift and the generation of theframe through the shifted pixels may be repeatedly performed.

In particular, the controller 130 may transmit the control signal C1when generation of the image frame by the image sensor 120 is completedand the image sensor 120 transmits a synchronization signal, whichinstructs transmission of the control signal C1 to the optical unit 110,thereto as a control signal C2. That is, a series of operationsincluding the pixel shift, the generation of the frame, and thesubsequent pixel shift may be performed through transmission andreception of the control signals C1 and C2 and synchronization thereof.

According to an embodiment, the image generation unit 200 may generatesuper-resolution image data having a high resolution, which has anintensity expressed using the following Equation 3 using the pluralityof image frames acquired as described above.

$\begin{matrix}{x_{\phi} = {\sum\limits_{k = 1}^{p}\left( {{D^{- 1}B_{k}^{- 1}A_{k}} - n_{k}} \right)}} & \left\lbrack {{Equation}3} \right\rbrack\end{matrix}$

Here, x_(ϕ) represents the intensity of the image data generated by theimage generation unit 200, 1≤k≤p, p represents the number of a pluralityof image frames having a spatial phase difference therebetween, whichare used to generate the image data, p being a positive integer of 2 ormore, and ϕ represents the degree of phase delay. When each of p imageframes includes I subframes, I=9, and p=4, ϕ may be expressed using thefollowing Equation 4.

$\begin{matrix}{\phi = {\arctan\left( \frac{Q_{3} - Q_{4}}{Q_{1} - Q_{2}} \right)}} & \left\lbrack {{Equation}4} \right\rbrack\end{matrix}$

Here, Q represents the intensity of the subframe.

In addition, in Equation 3 above, D⁻¹ represents the inverse matrix ofD, D represents the size of the pixel of the image sensor 120 (120-1 to120-H), B_(K) ⁻¹ represents the inverse matrix of B_(K), B_(K)represents optical characteristics with respect to depth information,n_(K) represents a noise component of a k^(th) image frame among p imageframes, and A_(K) represents the intensity of the k^(th) image frameamong the p image frames. A_(K) may be expressed using the followingEquation 5.A _(k) =DB _(k) x _(ϕ) +n _(k)  [Equation 5]

After step 20, the depth information extraction unit 300 extracts depthinformation about an object using the super-resolution image datagenerated by the image generation unit 200 (step 30).

According to an embodiment, when I=9 and p=4, the depth informationextraction unit 300 may calculate depth information, as expressed inEquation 6 below.

$\begin{matrix}{x = {\tan^{- 1}\frac{x_{\phi 1} - x_{\phi 3}}{x_{\phi 2} - x_{\phi 3}} \times \frac{c}{4\pi f}}} & \left\lbrack {{Equation}6} \right\rbrack\end{matrix}$

Here, x represents depth information, c represents a luminous flux, andf represents a frequency.

In the high-resolution image processing method according to theabove-described embodiment, image data having a high resolution isgenerated through Equation 3 using A_(k) in Equation 5 above.Thereafter, depth information x is obtained through Equation 6 aboveusing the image data having a high resolution obtained through Equation3.

In addition, the super-resolution image processing method according tothe embodiment may further include a step of calibrating depthinformation (hereinafter referred to as a ‘calibration step’). Thiscalibration step may be performed in step 30 described above, but theembodiment is not limited thereto.

Hereinafter, the calibration step according to the embodiment will bedescribed with reference to FIGS. 9, 10 (a), and 10(b). For convenienceof explanation, it is assumed that p=4 and I=9. Here, 1≤i≤I.

FIG. 9 is a flowchart for explaining an embodiment of the calibrationstep in the super-resolution image processing method according to theembodiment, and FIGS. 10(a) and (b) are waveform diagrams for helpingunderstanding of the calibration step shown in FIG. 9 . FIG. 10(a)illustrates a waveform diagram of an optical signal emitted toward anobject, and FIG. 10(b) illustrates a waveform diagram of an opticalsignal received by the image sensor 120 (120-1 to 120-H).

First, lens calibration is performed on first raw data (step 410). Here,the first raw data is data on four left subframes, among I subframes, ata first frequency f1 of a unit image frame, as shown in FIGS. 10(a) and(b). That is, the subframes from i=1 to i=4 correspond to the first rawdata. Lens calibration is an operation of calibrating accuratecoordinates with respect to lenses included in the optical unit 110(110-1 to 110-H).

After step 410 is performed, pixel calibration is performed on the firstraw data (step 420). Pixel calibration is an operation of calibratingvariation among the respective pixels of the image sensor 120 (120-1 to120-H).

After step 420 is performed, timing calibration is performed on thefirst raw data (step 430). Timing calibration is an operation ofcalibrating variation in the time at which each pixel of the imagesensor 120 (120-1 to 120-H) receives a signal.

After step 430 is performed, phase calibration is performed on the firstraw data (step 440). Phase calibration is an operation of calibratingvariation in the degree of phase delay ϕ in Equation 4, which differsamong the respective pixels of the image sensor 120 (120-1 to 120-H).

After step 440 is performed, the results of calibrating the first rawdata are stored (step 450).

After step 450 is performed, the above-described lens, pixel, timing,and phase calibrations are also performed on second raw data (steps 460to 490). Here, the second raw data is data on fifth to eighth subframesfrom the left, among the I subframes, at a second frequency f2, as shownin FIGS. 10(a) and (b). That is, the subframes from i=5 to i=8correspond to the second raw data. The first frequency f1 and the secondfrequency f2 are different from each other. For example, the firstfrequency f1 may be 80 MHz, and the second frequency f2 may be 60 MHz,but the embodiment is not limited thereto.

After step 450 is performed, lens calibration is performed on the secondraw data (step 460). After step 460 is performed, pixel calibration isperformed on the second raw data (step 470). After step 470 isperformed, timing calibration is performed on the second raw data (step480). After step 480 is performed, phase calibration is performed on thesecond raw data (step 490). After step 490 is performed, the results ofcalibrating the second raw data are stored (step 500).

After step 500 is performed, the above-described lens and pixelcalibrations are also performed on third raw data (steps 510 and 520).Here, the third raw data is data on the rightmost subframe, among the Isubframes, as shown in FIGS. 10(a) and (b). That is, the subframe of i=9corresponds to the third raw data.

After step 500 is performed, lens calibration is performed on the thirdraw data (step 510). After step 510 is performed, pixel calibration isperformed on the third raw data (step 520). After step 520 is performed,calibration for removing noise is performed on the third raw data (step530).

After step 530, the results of respectively calibrating the first,second and third raw data are synthesized (step 540). After step 540 isperformed, calibration is performed on the depth information (step 550).

The first, second and third raw data described above may be data in thestate after basic image signal processing (ISP) has been performed.

According to another embodiment, calibrations may be simultaneouslyperformed on the first to third raw data. That is, while steps 410 to440 are performed, steps 460 to 490 may be performed, and at the sametime, steps 510 to 530 may be performed.

Hereinafter, a super-resolution image processing method according to acomparative example and the super-resolution image processing methodaccording to the embodiment will be described with reference to theaccompanying drawings. For convenience of explanation, it is assumedthat p=4 and I=9.

FIG. 11(a) illustrates raw data, and FIG. 11(b) illustrates theintensity of an electric charge sensed and output by the image sensor120 (120-1 to 120-H).

The raw data shown in FIG. 11(a) is data on the k^(th) image frame,among a plurality of image frames having a spatial phase differencetherebetween, and the k^(th) image frame has nine (I=9) subframes.Accordingly, A_(ki) represents the intensity of the i^(th) subframe,among the subframes included in the k^(th) image frame.

Referring to FIGS. 11(a) and (b), A_(k2), A_(k3), A_(k4), and A_(k5)(i=2 to 5) represent an intensity raw data matrix (i.e. an intensitymatrix of subframes), which is acquired at a phase gated by first,second, third and fourth angles at the first frequency f1. A_(k6),A_(k7), A_(k8), and A_(k9) represent an intensity raw data matrix, whichis acquired at a phase gated by the first, second, third and fourthangles at the second frequency f2. For example, the first, second, thirdand fourth angles may be 0°, 90°, 180° and 270°, respectively, but theembodiment is not limited thereto.

In FIG. 11(b), the vertical axis represents the number p of electriccharges sensed and output by the image sensor 120 (120-1 to 120-H) ineach of nine subframes.

According to the super-resolution image processing method according tothe comparative example, depth information, which is expressed using thefollowing Equation 7, is generated using the intensity A_(k) of theframe, which is expressed using Equation 5 above, among a plurality ofimage frames having a spatial phase difference equivalent to a subpixelinterval therebetween.

$\begin{matrix}{y_{k} = {\tan^{- 1}\frac{A_{\phi 1} - A_{\phi 3}}{A_{\phi 2} - A_{\phi 3}} \times \frac{c}{4\pi f}}} & \left\lbrack {{Equation}7} \right\rbrack\end{matrix}$

Here, y_(k) represents depth information.

Thereafter, image data having a super resolution, which is expressedusing the following Equation 8, is generated using y_(k) in Equation 7.x=M′ _(k) ⁻¹ B′ _(k) ⁻¹ D′ ⁻¹ y _(k) −n _(K)  [Equation 8]

Here, x corresponds to super-resolution image data, 1≤k≤p, p representsthe number of image frames used to generate image data having a superresolution, M_(k)′⁻¹ represents the inverse matrix of M_(k)′, M_(k)′represents a depth point spread function (PSF), which may include blur,D′⁻¹ represents the inverse matrix of D′, D′ represents the size of thepixel of the image sensor, B_(k)′⁻¹ represents the inverse matrix ofB_(k)′, B_(k)′ represents optical characteristics with respect to depthinformation, n_(k) represents a noise component of the plurality ofimage frames, and A_(k) may be expressed using Equation 5 above.

FIGS. 12(a) to (c) are diagrams for explaining the super-resolutionimage processing method according to the comparative example, and FIG.12(a) illustrates the raw data and the intensity of an electric chargeshown in FIGS. 11(a) and (b).

That is, in the case of the comparative example, first depth informationy₁ about the first (k=1) image frame shown in FIG. 12(a) is obtained asshown in FIG. 12(b) using Equation 7, second depth information y₂ aboutthe second (k=2) image frame shown in FIG. 12(a) is obtained as shown inFIG. 12(b) using Equation 7, third depth information y₃ about the third(k=3) image frame shown in FIG. 12(a) is obtained as shown in FIG. 12(b)using Equation 7, and fourth depth information y₄ about the fourth (k=4)image frame shown in FIG. 12(a) is obtained as shown in FIG. 12(b) usingEquation 7.

Thereafter, the first to fourth depth information y₁ to y₄ may besubstituted into Equation 8 to obtain the super-resolution image shownin FIG. 12(c).

The super-resolution image processing method according to thecomparative example described above has a problem of an increased amountof computation because the computation process of Equation 7 forconverting image information into depth information is performed ptimes. In addition, even after depth information is extracted,additional modeling needs to be performed on M_(k)′, as expressed inEquation 8. In addition, the optical characteristics (e.g. B_(k)′) withrespect to depth information are more complicated than in the case ofthe image PSF.

On the other hand, in the case of the super-resolution image processingmethod according to the embodiment, super-resolution image data x_(ϕ) inEquation 3 above is generated using image data in Equation 5 in themiddle of the process of extracting distance information x from theimage frame through an I-time-of-flight (I-ToF) method.

Thereafter, depth information is extracted using the super-resolutionimage data x_(ϕ). Therefore, the super-resolution image processingmethod according to the embodiment is capable of solving a problem withthe resolution of I-ToF.

Consequently, like the comparative example, in the case in which imagedata having a high resolution is obtained after depth information isextracted, the amount of computation increases. The reason for this isthat the result of extracting depth information inherently containssupplementary data in addition to high-resolution image data, and thushas a large logic size. On the other hand, in the case of theembodiment, since high-resolution image data having a small logic sizeis obtained in advance before depth information is extracted, the amountof computation may be reduced.

FIGS. 13(a) to (c) are diagrams for explaining the super-resolutionimage processing method according to the embodiment. FIG. 13(a)illustrates four (P=4) image frames, and each of the four image frameshas the same intensity of electric charge as the raw data shown in FIGS.11(a) and (b).

In the super-resolution image processing method according to theembodiment, super-resolution image data x_(ϕ) shown in FIG. 13(c) isgenerated by substituting four image frames from the first (k=1) imageframe to the fourth (k=4) image frame into Equation 3.

Thereafter, depth information x is extracted by substituting thesuper-resolution image data x_(ϕ) into Equation 6, and therefore it isnot necessary to perform modeling on M_(k)′ in Equation 8, which is acomplicated PSF executed on depth data in the comparative example. Inaddition, the super-resolution image processing method according to theembodiment is capable of decreasing the complexity of an optical PSF forextracting depth information. In addition, in the case of thecomparative example, the computation (tan⁻¹) of Equation 7 is repeatedlyperformed, whereas in the case of the embodiment, the computation(tan⁻¹) of Equation 6 is performed only once. As a result, according tothe super-resolution image processing method according to theembodiment, the computation time required for image processing may beshortened, and image processing may be performed at a higher speed withthe same configuration.

Although only a limited number of embodiments have been described above,various other embodiments are possible. The technical contents of theabove-described embodiments may be combined into various forms as longas they are not incompatible with one another, and thus may beimplemented in new embodiments.

For example, an optical device (or an optical instrument) including theabove-described camera module may be implemented. Here, the opticaldevice may include a device that may process or analyze optical signals.Examples of the optical device may include camera/video devices,telescopic devices, microscopic devices, an interferometer, aphotometer, a polarimeter, a spectrometer, a reflectometer, anauto-collimator, and a lens-meter, and the embodiments may be applied tooptical devices that may include at least one of a solid lens or aliquid lens. In addition, the optical device may be implemented in aportable device such as, for example, a smartphone, a laptop computer,or a tablet computer. Such an optical device may include a cameramodule, a display unit configured to output an image, and a body housingin which the camera module and the display unit are mounted. Acommunication module, which may communicate with other devices, may bemounted in the body housing of the optical device, and the opticaldevice may further include a memory unit capable of storing data.

It will be apparent to those skilled in the art that various changes inform and details may be made without departing from the spirit andessential characteristics of the disclosure set forth herein.Accordingly, the above detailed description is not intended to beconstrued to limit the disclosure in all aspects and to be considered byway of example. The scope of the disclosure should be determined byreasonable interpretation of the appended claims and all equivalentmodifications made without departing from the disclosure should beincluded in the following claims.

Mode for Invention

Various embodiments have been described in the best mode for carryingout the disclosure.

INDUSTRIAL APPLICABILITY

A camera module and a super-resolution image processing method thereofaccording to embodiments may be used in camera/video devices, telescopicdevices, microscopic devices, an interferometer, a photometer, apolarimeter, a spectrometer, a reflectometer, an auto-collimator, alens-meter, a smartphone, a laptop computer, a tablet computer, etc.

The invention claimed is:
 1. A camera module, comprising: an imageacquisition unit configured to acquire a plurality of image frameshaving a spatial phase difference therebetween; an image generation unitconfigured to generate image data having a resolution higher than aresolution of each of the plurality of image frames using the pluralityof image frames; and a depth information extraction unit configured toextract depth information about an object using the image data, whereina logic size of a result of extracting the depth information is greaterthan a logic size of the image data generated by the image generationunit, the result of extracting the depth information includingsupplementary data.
 2. A camera module, comprising: an image acquisitionunit configured to acquire a plurality of image frames having a spatialphase difference therebetween; an image generation unit configured togenerate image data having a resolution higher than a resolution of eachof the plurality of image frames using the plurality of image frames;and a depth information extraction unit configured to extract depthinformation about an object using the image data, wherein the imageacquisition unit comprises: an optical unit configured to change a pathalong which light for the object travels; an image sensor configured tosense light incident along different paths; and a controller configuredto control the optical unit and the image sensor, wherein the pluralityof image frames corresponds to results sensed in sequence by the imagesensor, and wherein the image generation unit generates the image datausing an intensity obtained below on each of the plurality of imageframes:D ⁻¹ B _(k) ⁻¹ A _(k) −n _(k) where 1≤k≤p, p represents a number of theimage frames used to generate the image data, D⁻¹ represents an inversematrix of D, D represents a size of a pixel of the image sensor, B_(K)⁻¹ represents an inverse matrix of B_(k), B_(k) represents opticalcharacteristics with respect to the depth information, n_(k) representsa noise component of a k^(th) image frame among p image frames, andA_(k) represents an intensity of the k^(th) image frame among the pimage frames, and is as follows:A _(k) =DB _(k) x _(ϕ) +n _(k), where x_(ϕ) represents an intensity ofthe image data and ϕ represents a degree of phase delay.
 3. The cameramodule according to claim 2, wherein the image generation unit generatesthe image data having the intensity as follows:${x_{\phi} = {\sum\limits_{k = 1}^{p}\left( {{D^{- 1}B_{k}^{- 1}A_{k}} - n_{k}} \right)}}.$4. The camera module according to claim 2, wherein the depth informationextraction unit calculates the depth information as follows:$x = {\tan^{- 1}\frac{x_{\phi 1} - x_{\phi 3}}{x_{\phi 2} - x_{\phi 3}} \times \frac{c}{4\pi f}}$where x represents depth information, c represents a luminous flux, andf represents a frequency.
 5. The camera module according to claim 2,wherein the image acquisition unit comprises: a plurality of opticalunits having respectively different paths along which light for theobject travels; and a plurality of image sensors configured to senselight incident through the plurality of optical units, and wherein theplurality of image frames corresponds to results sensed by the pluralityof image sensors.
 6. The camera module according to claim 2, wherein theplurality of image frames has a spatial phase difference correspondingto an interval smaller than a pixel interval therebetween.
 7. The cameramodule according to claim 2, wherein the controller changes the pathalong which light travels from the optical unit so that the plurality ofimage frames, sequentially sensed and output by the image sensor, has aspatial phase difference corresponding to an interval smaller than apixel interval therebetween.
 8. A super-resolution image processingmethod of a camera module, comprising: acquiring a plurality of imageframes having a spatial phase difference therebetween; and extracting ofdistance information about an object from the plurality of image framesthrough an I-time-of-flight (I-ToF) method, wherein image data having aresolution higher than a resolution of each of the plurality of imageframes is generated using the plurality of image frames, during aprocess of extracting the distance information, wherein the distanceinformation includes a depth information, and wherein the depthinformation about the object is extracted using the generated imagedata.
 9. The super-resolution image processing method according to claim8, wherein the plurality of image frames is synthesized to generate theimage data corresponding to an image acquired by a 2N×2M pixel arrayrather than by an N×M pixel array, an image sensor having the N×M pixelarray.
 10. A super-resolution image processing method of a cameramodule, comprising: (a) acquiring a plurality of image frames having aspatial phase difference therebetween; (b) generating image data havinga resolution higher than a resolution of each of the plurality of imageframes using the plurality of image frames; and (c) extracting depthinformation about an object using the image data, wherein step (b)comprises obtaining the image data using an intensity obtained below oneach of the plurality of image frames:D ⁻¹ B _(k) ⁻¹ A _(k) −n _(k) where 1≤k≤p, p represents a number of theimage frames used to generate the image data, D⁻¹ represents an inversematrix of D, D represents a size of a pixel of an image sensor obtainingeach of the image frames, B_(k) ⁻¹ represents an inverse matrix ofB_(k), B_(k) represents optical characteristics with respect to thedepth information, n_(k) represents a noise component of a k^(th) imageframe among p image frames, and A_(k) represents an intensity of thek^(th) image frame among the p image frames, and is as follows:A _(k) =DB _(k) x _(ϕ) +n _(k) wherein x_(ϕ) represents an intensity ofthe image data and ϕ represents a degree of phase delay.
 11. Thesuper-resolution image processing method according to claim 10, whereinthe intensity of the image data is as follows:${x_{\phi} = {\sum\limits_{k = 1}^{p}\left( {{D^{- 1}B_{k}^{- 1}A_{k}} - n_{k}} \right)}}.$12. The super-resolution image processing method according to claim 10,wherein step (c) comprises: obtaining the depth information as follows:$x = {\tan^{- 1}\frac{x_{\phi 1} - x_{\phi 3}}{x_{\phi 2} - x_{\phi 3}} \times \frac{c}{4\pi f}}$where x represents depth information, c represents a luminous flux, andf represents a frequency.
 13. The super-resolution image processingmethod according to claim 10, wherein step (a) comprises: changing apath along which light for the object travels; and sensing lightincident along different paths in sequence to acquire the plurality ofimage frames.
 14. The super-resolution image processing method accordingto claim 13, wherein the plurality of image frames has a spatial phasedifference corresponding to an interval smaller than a pixel intervaltherebetween, by changing the path along which light travels.
 15. Thesuper-resolution image processing method according to claim 10, whereinstep (a) comprises: sensing light for the object simultaneously indifferent paths to acquire the plurality of image frames.
 16. Thesuper-resolution image processing method according to claim 10,comprising performing calibrations on the result of extracting the depthinformation.
 17. The super-resolution image processing method accordingto claim 16, wherein each of the plurality of image frames includes aplurality of subframes, wherein the calibrations are performed on eachof a plurality of raw data, the plurality of raw data being data on theplurality of subframes, and wherein the calibrations include at leasttwo of lens calibration, pixel calibration, timing calibration, andphase calibrations.
 18. The super-resolution image processing methodaccording to claim 17, wherein the calibrations include a calibrationperformed on a last raw data for removing noise, the last raw datacorresponding to a raw data on which the calibrations are performedlast, among the plurality of raw data.
 19. The super-resolution imageprocessing method according to claim 18, comprising synthesizing resultsof respectively calibrating the plurality of raw data after removing thenoise.
 20. The super-resolution image processing method according toclaim 19, wherein the calibrations include a calibration performed onthe depth information after performing the synthesizing.